# Lightweight Pretraining

Arsh LLM is an open-source large language model designed for research, pretrained on the olmo mixed dataset using a T4 GPU, with a total training time of approximately 4-5 days.

A language model scaled down from Mistral 7B to 248 million parameters, designed for text generation tasks and suitable for downstream task fine-tuning.

SEW-tiny is a compressed and efficient speech pretraining model developed by ASAPP Research, pretrained on 16kHz sampled speech audio, suitable for various downstream speech tasks.

A lightweight BERT model pretrained using knowledge distillation techniques, with a hidden layer dimension of 256 and 4 attention heads, suitable for masked language modeling tasks.

A Chinese multimodal image captioning model fine-tuned on the AIC-ICC Chinese image caption dataset, based on the Mengzi-Oscar pretrained model

Pretrained model for Arabic NLP tasks, trained on a reduced-scale (1/16) Modern Standard Arabic (MSA) dataset